Docs LLM accessibility: add llms.txt, allow copying and viewing pages as markdown#581
Docs LLM accessibility: add llms.txt, allow copying and viewing pages as markdown#581
llms.txt, allow copying and viewing pages as markdown#581Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
Amplify deployment status
|
…t the markdown output for pages
|
I had Claude Code compare the preview site and current docs site, and it concluded that the preview site provides more information, particularly by not hiding all tab items but the selected one in |
ptgott
left a comment
ptgott
left a comment
There was a problem hiding this comment.
This is a partial review (I still need to look at the PageActions code, src/utils/markdown.ts, and docusaurus.config.ts). I had a nitpicky question about how we organize the plugin code and, more importantly, a question about hardcoding descriptions of docs sections and Teleport as a whole. Thanks for working on this!
| import rehypeProcessCustomComponentsForMarkdown from "./rehype-process-custom-components"; | ||
|
|
||
| // Define the sections for the llms.txt index file, including their routes and descriptions | ||
| const sections = [ |
There was a problem hiding this comment.
Is there a way to generate this from the docs content or check that it is correct? With the current approach, the LLM version of the docs will eventually drift from the human version of the docs.
| enableDescriptions: true, | ||
| siteTitle: "Teleport documentation", | ||
| siteDescription: | ||
| "Teleport is an identity-based access platform that secures servers, Kubernetes clusters, databases, internal applications, and desktops using short-lived certificates, detailed audit logging, and fine-grained role-based access controls tied to your SSO provider (e.g., Okta, GitHub, Google Workspace).", |
There was a problem hiding this comment.
I'm also wondering if we can generate the description. If we change our top-level messaging strategy, for example, I can see a situation in which we forget to update this and give LLMS an outdated impression of the product.
There was a problem hiding this comment.
Would it make sense to combine this with rehype-process-custom-components.ts? Then we could name the plugin after the purpose—preparing the HTML for LLMS—rather than whether the components are custom or not.
There was a problem hiding this comment.
Could we combine this with the other rehype LLM preparation plugins?
There was a problem hiding this comment.
Same with this one: would it make sense to consolidate visitParents calls and have a single plugin?
If that's too unwieldy and there's no performance difference, I can see the case for a few small plugins in the llms directory.
2 participants
The first iteration of features dedicated to providing LLMs better context of the docs site.
New features:
/docs/get-startedgets a markdown version in/docs/get-started.md)llms.txtfile (available at/docs/llms.txt)A new Docusaurus plugin,
@signalwire/docusaurus-plugin-llms-txt, is installed for the task of generating the markdown pages and thellms.txtfile. The plugin is configured in a new file/server/llms/index.ts.List of changes:
@signalwire/docusaurus-plugin-llms-txtto the list of dependencies inpackage.json@signalwire/docusaurus-plugin-llms-txtplugin to the Docusaurus plugins configuration indocusaurus.config.ts@signalwire/docusaurus-plugin-llms-txtplugin in/server/llms/index.tsrehypeplugins to/server/llms/, which are used in the@signalwire/docusaurus-plugin-llms-txtplugin:rehype-process-custom-components.ts: Used to process custom components for cleaner markdown outputrehype-remove-heading-anchor-links.ts: Prevents the automatically created heading anchor links from appearing in the markdown outputrehype-remove-irrelevant-components.ts: Prevents components not useful for AI to appear in the markdown outputrehype-reposition-h1.ts: Ensures the page title is correctly applied to the markdown version of the pagerehypeplugins.rehypeunit tests/server/fixtures/llmsPageActionscomponent to accommodate the design changes found in Figmaunist, which caused some unit tests to fail (/server/unist-augment.d.ts)